Monitoring of customer consumption activity and management based on monitoring

ABSTRACT

A monitoring-based service providing system includes a table monitoring terminal associated with one of a plurality of tables in a venue, a central control terminal, and an order terminal associated with said one of the plurality of tables. The central control terminal includes a communication interface communicable with the table monitoring terminal, and a controller. The controller is configured to compare the image data received at first timing with the image data received at second timing after the first timing, determine a remaining amount of the consumable item based on a comparison result of the image data received at the first and second timing, and generate recommendation information upon the determining remaining amount decreasing to a predetermined threshold, and control the second communication interface to transmit the generated recommendation information. The order terminal includes a display, on which a recommendation screen is displayed based on the recommendation information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/289,381, filed on Feb. 28, 2019, which is based upon and claims thebenefit of priority from Japanese Patent Application No. 2018-037865,filed on Mar. 2, 2018, the entire contents of each of which areincorporated herein by reference.

FIELD

Embodiments described herein relate generally to a system to performmonitoring of a customer consumption activity and management based onthe monitoring.

BACKGROUND

In a related art, an order processing system, in which a customer ordersa desired or selected consumable item (food, alcohol and beverage) byoperating an order terminal, is utilized in a restaurant. The orderprocessing system includes a function of offering a recommended item forpromotion.

In such a menu order system, it may be necessary to recommend an item atan appropriate timing according to the customer based on a consumptionstate of a consumable item.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an order processing systemaccording to an embodiment.

FIG. 2 is a block diagram illustrating an essential part of a hallcontroller.

FIG. 3 is a block diagram illustrating an essential part of an orderreception terminal.

FIG. 4 is a block diagram illustrating an essential part of a tablerobot.

FIG. 5 is a diagram illustrating appearances of the order receptionterminal and the table robot.

FIG. 6 is a flowchart illustrating an operation according to a firstembodiment.

FIG. 7 is a flowchart illustrating comparison processing performed bythe hall controller.

FIG. 8 is a diagram conceptually illustrating a first image and a secondimage captured by the table robot.

FIG. 9 is a diagram conceptually illustrating recommendation informationextracted by performing a recommendation information extractionprocessing.

FIG. 10 is a diagram illustrating a recommendation menu screen.

FIG. 11 is a flowchart illustrating an operation according to a secondembodiment.

FIG. 12 is a diagram illustrating a screen showing a seat status.

FIG. 13 is a diagram illustrating a detail screen (details of the seatstatus).

FIG. 14 is a flowchart illustrating an operation according to a thirdembodiment.

DETAILED DESCRIPTION

According to an embodiment, a monitoring-based service providing systemincludes a table monitoring terminal associated with one of a pluralityof tables in a venue, a central control terminal, and an order terminalassociated with said one of the plurality of tables. The tablemonitoring terminal includes a camera, a driving mechanism configured tochange a direction of the camera, a first communication interface, and afirst controller. The first controller is configured to control thedriving mechanism such that the camera is directed to at least aconsumable item on an associated table, and control the firstcommunication interface to transmit image data of an image including aconsumable item captured by the camera. The central control terminalincludes a second communication interface communicable with the firstcommunication interface, and a second controller. The second controlleris configured to compare the image data received by the secondcommunication at first timing with the image data received by the secondcommunication interface at second timing after the first timing,determine a remaining amount of the consumable item based on acomparison result of the image data received at the first and secondtiming, and generate recommendation information upon the determiningremaining amount decreasing to a predetermined threshold, and controlthe second communication interface to transmit the generatedrecommendation information. The order terminal includes a thirdcommunication interface communicable with the second communicationinterface, a display, and a third controller configured to control thedisplay to display a recommendation screen based on the recommendationinformation received by the third communication interface.

Hereinafter, an embodiment is described with reference to theaccompanying drawings.

FIG. 1 is a block diagram illustrating an order processing systemaccording to the embodiment. The order processing system is provided andoperated, for example, in a restaurant.

In the order processing system, an order station 10, a Point of Sales(POS) terminal 12, a kitchen printer 14, a hall controller 16, and atleast one access point (AP) 18 (18-1, . . . , 18-m) are connected toeach other through a network 19 such as a Local Area Network (LAN). Inaddition, a tablet terminal 16K, a plurality of handy terminals 20(20-1, . . . , 20-n), a plurality of order reception terminals 22 (22-1,. . . , 22-x) and a plurality of table robots 24 (24-1, . . . , 24-x),provided on respective tables in a restaurant, and at least one floorrobot 26 (26-1, . . . , 26-y) are connected to the network 19 through awireless communication with the access point 18.

The order station 10 transmits a cooking instruction to the kitchenprinter 14 based on order information received from the handy terminal20 or the order reception terminal 22. If a delivering completion isreceived from the handy terminal 20, the order station 10 transmits theorder information to the POS terminal 12 as sales information.

The POS terminal 12 is provided, for example, on a checkout counter, andis used to settle a cost to be paid for eating and drinking by a guest.The POS terminal 12 registers and processes the sales information, toperform settlement processing for every transaction. The salesinformation includes information of a sales specification of items forevery customer or every table.

The kitchen printer 14 is arranged, for example, at a cooking place or akitchen. The kitchen printer 14 prints an order slip on which foodand/or beverages including alcohol (menu item) ordered by the customerand a table identification number are indicated, based on the orderinformation received from the order station 10 to output a cookinginstruction.

The hall controller 16 is a computer operating as an informationprocessing apparatus that performs processing for controlling andmanaging the floor robot 26. The hall controller 16 manages a situationand/or a state (hereinafter referred to collectively as a situation) ofthe customer in the restaurant to control respective devices included inthe order processing system according to the situation of the customer.The hall controller 16 manages, for example, a situation of a tablewhere the customer takes a seat, customer information (number ofpersons, gender, age, job, and the like), an order history, an occupiedtime (time elapsed from taking the seat, a remaining time up to asetting end time for a prix-fix menu), and the like. The hall controller16 displays a screen for showing the situation of the customer.Employees, e.g., waitress and waiter, of the restaurant look at thescreen of the hall controller 16. The hall controller 16 inputs imagingdata indicating an area, including a table on which at least an item(beverage, food, and the like) is placed, as an image capturing area,and detects a remaining quantity of the item by performing an imageprocessing based on the imaging data. The hall controller 16 has afunction of outputting recommendation information used to recommend asuitable item to the customer at an appropriate timing based on theremaining quantity of the item determined according to a result of theimage processing. The hall controller 16 inputs imaging data captured bya camera provided, for example, in the table robot 24 (or the orderreception terminal 22) on the table. The imaging data may be any of astill image or a moving image (video).

The tablet terminal 16K is used as an input/output device of the hallcontroller 16. The tablet terminal 16K communicates with the hallcontroller 16 through the access point 18 and the network 19. Meanwhile,the tablet terminal 16K may directly perform the wireless communicationwith the hall controller 16 without connecting through the access point18. The number of tablet terminals 16K is not limited to one, and aplurality of tablet terminals 16K may be provided.

Although FIG. 1 shows the hall controller 16 implemented by a specificcomputer connected to the network 19, processing of the hall controller16 described below may be performed by the tablet terminal 16K.Therefore, in the present embodiment, any one of a form, in which thehall controller 16 is implemented by the computer connected to thenetwork 19, and a form, in which the hall controller 16 is implementedby the tablet terminal 16K can be adopted. Furthermore, the processingmay be performed by not only one hall controller 16 (or the tabletterminal 16K) as described above but also the hall controller 16 incooperation with another device (a plurality of computers or the orderstation 10).

The access point 18 is a wireless communication device for controllingthe wireless communication with the tablet terminal 16K, the handyterminal 20, the order reception terminal 22, the table robot 24, andthe floor robot 26. At least one access point 18 is arranged on, forexample, a ceiling in the restaurant.

The handy terminal 20 is used by the clerk. The handy terminal 20performs an input of the order, check-in processing of registering acustomer who visits the restaurant, an input of completion of deliveringthe item ordered by the customer, and an output of a message accordingto a notification from the hall controller 16. The handy terminal 20transmits the order information according to the input order and iteminformation indicative of the item, of which delivering is completed, tothe order station 10. The handy terminal 20 transmits variousnotifications to the hall controller 16 in response to the operation bythe clerk.

The order reception terminal 22 (22-1, . . . , 22-x) is placed, forexample, on each table T (T−1, . . . , T-x) in the restaurant. The orderreception terminal 22 displays a menu and receives the order of the item(menu item) from the customer. The order reception terminal 22 transmitsthe input order information to the order station 10. The order receptionterminal 22 displays a recommendation item screen according to therecommendation information output from the hall controller 16.

The table robot 24 (24-1, . . . , 24-x) is disposed, for example, oneach table T (T−1, . . . , T-x) in the restaurant. The table robot 24performs communication with the customer. The table robot 24 inputs theimaging data indicating an area, including the table on which at leastan item (beverage, food, or the like) is placed, which is captured bythe camera as the image capturing area and transmits the imaging data tothe hall controller 16. In addition, the table robot 24 has a functionof recognizing voice data input from a microphone, to perform a simpleconversation with the customer and reception of an item order with voiceof the customer. The table robot 24 includes a voice output function ofoutputting a message to the customer from the speaker, to perform anoutput of a response message, corresponding to the voice recognizedthrough the voice recognition function, for conversation with thecustomer, and an output of a message according to the notification fromthe hall controller 16 or the order station 10.

Meanwhile, in FIG. 1, the order reception terminal 22 and the tablerobot 24 are implemented by separate devices. However, it is possiblethat the order reception terminal 22 and the table robot 24 are providedas one device.

The floor robot 26 is used to deliver the item to each table and tocollect (bus) articles (tableware), such as dishes, glasses, and waste,from each table. The floor robot 26 is provided on at least one placingtable on which the tableware and the like are placed. The floor robot 26includes an autonomous travel function of traveling to a location of adesignated target table along a path predetermined based on the locationof the table in the restaurant. The floor robot 26 includes, forexample, an anti-collision sensor, to avoid obstacles while traveling tothe table.

The floor robot 26 includes a wireless communication function throughthe access point 18. The floor robot 26 receives, for example, aninstruction input from the order reception terminal 22 by the customer,an instruction from the hall controller 16 according to the situation ofthe customer, or an instruction input through the operation by theemployee, to perform operation according to the instruction. Inaddition, the floor robot 26 notifies locational data indicative of acurrent location thereof to the hall controller 16 through the accesspoint 18 while traveling. The hall controller 16 displays, for example,on a floor map displaying the table arrangement, information indicatingthe current location (operational situation) of the floor robot 26 on apath along which the floor robot 26 travels based on the locational datareceived from the floor robot 26.

The floor robot 26 includes an input/output device, such as a touchpanel, a person authentication function based on an image captured bythe camera, a voice recognition function of voice data input from amicrophone, and a voice output function of outputting a message to thecustomer or the clerk from the speaker to perform communication with thecustomer or the clerk.

FIG. 2 is a block diagram illustrating an essential part of the hallcontroller 16 according to the present embodiment.

The hall controller 16 is implemented by a specific computer. The hallcontroller 16 includes a Central Processing Unit (CPU) 16A, a Read OnlyMemory (ROM) 16B, a Random Access Memory (RAM) 16C, a storage device16D, a display device 16E, an input device 16F, a touch panel 16G, and acommunication device 16H.

The CPU 16A executes a program stored in the ROM 16B or the storagedevice 16D to control above respective devices of the hall controller16. The program, executed by the CPU 16A, includes an informationprocessing program to operate as the information processing apparatus,in addition to a basic program (Operating System (OS)). The CPU 16Aexecutes the information processing program to manage the situation ofthe customer in the restaurant. The CPU 16A has an input function ofinputting the imaging data transmitted from the table robot 24 (or theorder reception terminal 22) through the communication device 16H, adetection function of detecting a remaining quantity of the item(beverage or food) by performing the image processing on the inputimaging data, and an output function of outputting the recommendationinformation to recommend an item to the customer at an appropriatetiming based on the remaining quantity of the item. In addition, the CPU16A performs a recommendation information extraction process ofdetermining the recommendation information according to the customerbased on, for example, the situation of the table of which the seat istaken, management of the customer information (number of persons,gender, age, job, and the like), management of the order history, theoccupied time (time elapsed from taking the seat, and the remaining timeup to the setting end time of prix fix menu), and the like.

The ROM 16B stores the program executed by the CPU 16A. The RAM 16Cforms various work memory areas. The storage device 16D includes, forexample, a Hard disk drive (HDD), and a Solid State Drive (SSD) to storevarious programs and various data.

The display device 16E is, for example, a Liquid Crystal Display (LCD).The input device 16F is, for example, a pointing device such as akeyboard or a mouse. The touch panel 16G performs display and inputthrough touch operation thereon.

The communication device 16H controls communication with, for example,respective devices through the network 19. The communication device 16Hmay include a wireless communication function.

FIG. 3 is a block diagram illustrating an essential part of the orderreception terminal 22 according to the present embodiment.

The order reception terminal 22 is implemented by, for example, aspecific tablet-type computer. The order reception terminal 22 includesa CPU 22A, a ROM 22B, a RAM 22C, a storage device 22D, a touch panel22G, and a communication device 22H.

The CPU 22A executes a program stored in the ROM 22B or the storagedevice 22D to control operation sections of the order reception terminal22. The program executed by the CPU 22A includes an order receptionprogram (application program) to operate as the order reception terminal22, in addition to the basic program (Operating System (OS)). The CPU22A executes the order reception program to perform processing ofdisplaying a menu screen, receiving an input operation by the customer,transmitting the order information to the order station 10 according tothe item selected from the menus screen through the input operation bythe customer, notifying (displaying or voice outputting) of arecommended item according to the recommendation information transmittedfrom the hall controller 16 (or the order station 10). The CPU 22Adisplays a screen (Graphical User Interface (GUI)) according to therespective processing described above on the touch panel 22G.

The ROM 22B stores program executed by the CPU 22A. The RAM 22C formsvarious work memory areas. The storage device 22D includes, for example,the HDD, and the SSD, to store the various programs and various data.The touch panel 22G performs the display and the input through touchoperation. The communication device 22H controls the wirelesscommunication with the access point 18.

Meanwhile, the order reception terminal 22 may include a similarfunction as the table robot 24 described below, and thus the table robot24 can be omitted. In this case, the camera, the microphone, and thespeaker are provided in the order reception terminal 22, and thuscontrol functions, corresponding to the respective devices, which aresimilar to those of the table robot 24 described below, are provided.

FIG. 4 is a block diagram illustrating an essential part of the tablerobot 24 according to the present embodiment.

The table robot 24 includes a function of a specific computer. The tablerobot 24 includes a CPU 24A, a ROM 24B, a RAM 24C, a storage device 24D,a microphone 24E, a speaker 24F, a camera 24G, a communication device24H, and a driving mechanism 24L.

The CPU 24A executes a program stored in the ROM 24B or the storagedevice 24D to control the respective operation sections of the tablerobot 24. The program executed by the CPU 24A includes a control programto operate as the table robot 24, in addition to a basic program(Operating System (OS)). The CPU 24A executes the control program tocontrol an operation of the camera 24G and performs an output functionof outputting the imaging data captured by the camera 24G to the hallcontroller 16 through the communication device 24H. In addition, the CPU24A executes the control program to control an operation forcommunication with the customer. The communication with the customerincludes communication through the voice (conversation), communicationthrough the operation, and the like. The CPU 24A performs a voicerecognition function of recognizing voice data input from the microphone24E, and a function of outputting voice from the speaker 24F tocommunicate with the customer through voice (conversation).

The CPU 24A recognizes voice from the customer to perform theconversation, and to receive order of the item from the customer, usingthe function of recognizing voice. In the function of recognizing voice,for example, a predetermined voice command (word or sentence) can berecognized and thus processing according to the voice command can beperformed. In addition, the CPU 24A uses the voice output function ofoutputting a response message answering to the voice, used to performthe communication, which is recognized through the voice recognitionfunction, and to output a message responding to a notification from thehall controller 16 or the order station 10.

The ROM 24B stores program executed by the CPU 24A. The RAM 24C formsvarious work memory areas. The storage device 24D includes, for example,the HDD, and the SSD, to store various programs and various data.

The camera 24G performs an image capturing under the control of the CPU24A, to output imaging data. The camera 24G captures an image in a stillform or a moving form (video). The communication device 24H controlswireless communication with the access point 18.

The CPU 24A operates the driving mechanism 24L. The table robot 24includes a plurality of housings (for example, an upper housing and alower housing), and a part of the housings can be moved by the drivingmechanism 24L. The CPU 24A drives, for example, for the communicationwith the customer, the driving mechanism 24L in accordance with a normalstate, conversation with voice, order of the item, and output timings ofvarious message voice. The CPU 24A also drives the driving mechanism 24Lsuch that a direction of the camera 24G is changed to change the imagecapturing area of the camera 24G.

The driving mechanism 24L has a mechanical structure to move at least apart of the housings of the table robot 24.

FIG. 5 is a diagram illustrating appearances of the order receptionterminal 22 and the table robot 24 according to the present embodiment.FIG. 5 shows an arrangement of the order reception terminal 22 and thetable robot 24 disposed side by side on a table.

Basically, the order reception terminal 22 is placed in a cradle on thetable. The order reception terminal 22 can be taken from the cradle andused in a state of being held by the customer, if necessary.

The table robot 24 is used to perform communication with the customer,and thus it is desirable that the table robot 24 is positioned at alocation close to the customer who takes the seat at the table. Inaddition, the table robot 24 performs the image capturing of an areaincluding the table on which at least an item (beverage, food, or thelike) is placed as the image capturing area. For example, the tablerobot 24 may capture not only an image of the table on which the item isplaced but also an image of an area including the customer who takes theseat at the table as the image capturing area. Therefore, the tablerobot 24 is placed at a location at which the camera 24G of the tablerobot 24 can capture an image of the target image capturing area.

The camera 24G of the table robot 24 shown in FIG. 5 is mounted, forexample, in the upper housing, and also performs the image capturingwhile the upper housing moves vertically and horizontally in aprescribed range by the driving mechanism 24L. Therefore, an image of awider area can be captured as the image capturing area compared with acase in which the image capturing direction of the camera 24G is fixed.

Subsequently, an operation of the order processing system according tothe present embodiment is described.

In the order processing system, check-in processing for registering acustomer who visits the restaurant is performed if the customer takes aseat at one of the tables. In the check-in processing, registration of atable number, and inputs of the customer information (number of persons,gender, age, job, and the like), an ordered course menu (a plurality ofitems to be served and a time are predetermined (prix fix menu)), andthe like are performed. For example, the customer information may beinput through the operation on the handy terminal 20 carried by theclerk or may be input through the operation on the order receptionterminal 22 by the customer.

In addition, a video image (imaging data) captured by the camera of thetable robot 24 is transmitted to the hall controller 16, the hallcontroller 16 (or the tablet terminal 16K) displays the video image, andthus the customer information may be input by a clerk who checks thevideo image. Furthermore, the hall controller 16 performs personauthentication processing on an image acquired by photographing thecustomer to detect number of persons, gender, age, job, and the like anda result of the detection may be input as the customer information.

When the customer orders an item after the customer takes the seat atthe table, the customer operates the order reception terminal 22 ororders the item to the clerk who carries the handy terminal 20. Theorder reception terminal 22 displays the menu on the touch panel 22G,and transmits the order information of the item selected from the menuthrough the touch operation on the touch panel to the order station 10.The menu is displayed in such a manner that a usual menu including allitems categorized and a recommendation menu including specificrecommendation items prepared in advance are switched in response to theinput operation by the customer.

The ordered item is delivered to the table by the clerk, i.e., waiter orwaitress, or the floor robot 26. For example, if a table, i.e., deliverydestination, is designated after the ordered items, such as a glass(beverage) or a dish (food) is placed on the placing table, the floorrobot automatically travels to the designated table deliver the item.

The hall controller 16 checks the situation of the table at which thecustomer takes the seat, using the video image captured by the camera24G of the table robot 24. In the present embodiment, the hallcontroller 16 detects progress in eating and drinking of the delivereditem (beverage or food), that is, a situation in which the item consumedby the customer based on the captured video image, and outputs therecommendation information to recommend an item to the customer at anappropriate timing. Hereinafter, processing (according to a firstembodiment, a second embodiment, and a third embodiment) of outputtingthe recommendation information by the hall controller 16 is described.

First Embodiment

FIG. 6(A) is a flowchart illustrating processing performed by the tablerobot 24 and the order reception terminal 22 according to the firstembodiment, and FIG. 6(B) is a flowchart illustrating processingperformed by the hall controller 16 according to the first embodiment.In the description below, in the flowchart of FIG. 6(A), Act A1 to ActA5 are described as operations of the table robot 24 and Act A6 to ActA11 are described as operations of the order reception terminal 22. Ifthe order reception terminal 22 performs a function of the table robot24, the processing of the flowchart illustrated in FIG. 6(A) isperformed by the order reception terminal 22.

After the item is delivered to the table, the table robot 24 performscapturing of an image (first image) of an area including at least thetable on which the item is placed by a camera 24G, as an image capturingarea (Act A1). The captured first image shows an initial state of theitem not yet consumed by the customer. The table robot 24 adds, forexample, data indicating the table number to the imaging data of thefirst image and transmits the imaging data of the first image to thehall controller 16 through the communication device 24H (Act A2).

When the CPU 16A of the hall controller 16 receives the imaging data ofthe first image transmitted from the table robot 24 (Yes in Act B1), theCPU 16A stores the imaging data in the RAM 16C or the storage device 16Din association with the table number.

The CPU 16A monitors elapse of prescribed time preset for the table ofwhich the first image is received. If it is determined that theprescribed time elapses (Yes in Act B2), the CPU 16A transmits an imagetransmission request to the relevant table robot 24 (Act B3).

The prescribed time may be one fixed time predetermined or may be variedaccording to a time which elapses after the check-in processing isperformed for the customer. If the prescribed time is varied, theprescribed time is set to a short time (for example, three minutes) in atime zone (for example, 0 to 15 minutes), for example, in which eatingand drinking pace of the customer is usually fast, immediately after thestart. The closer the end at which an item order pace generally becomesslow, the longer the prescribed time in a step-wise. For example, theprescribed time may be set to 5 minutes in a time zone of 15 to 30minutes and may be set to 10 minutes in a time zone of 30 to 60 minutes.In this way, processing loads may be reduced in such a way that arecommendation item menu is offered to the customer at more appropriatetiming or in such a way that a frequency in the image processing beingnecessary is reduced in a time zone in which the item order pace becomesslow.

When the CPU 24A of the table robot 24 receives the image transmissionrequest from the hall controller 16 (Yes in Act A3), for example, theCPU 24A performs capturing of an image (second image) of the same imagecapturing area as in a case in which the first image is captured, by thecamera 24G (Act A4). The table robot 24 adds the data indicating thetable number to the imaging data of the second image and transmits theresulting imaging data of the second image to the hall controller 16(Act A5). The second image is captured such that the same imagecapturing area as in the case in which the first image is captured iscaptured again after the prescribed time elapses from the capturing ofthe first image.

When the CPU 16A of the hall controller 16 receives the imaging data ofthe second image transmitted from the table robot 24 (Yes in Act B4),the CPU 16A stores the imaging data of the second image in the RAM 16Cor the storage device 16D in association with the table number, andperforms comparison processing for detecting a change in a state(remaining quantity of the item) by comparing the first image with thesecond image.

FIG. 7 is a flowchart illustrating the comparison processing performedby the hall controller 16.

First, the CPU 16A determines whether or not the first image is alreadyreceived and object detection described below is completed for the firstimage. If the first image is not yet received (No in Act S1), the CPU16A detects an object image (first object) corresponding to the itemprovided to the customer from the first image (Act S2), and stores thefirst object in the RAM 16C or the storage device 16D (Act S3).

FIG. 8(A) is a diagram conceptually illustrating the first image. Asillustrated in FIG. 8(A), there is a case in which the first imagecaptured by the camera 24G of the table robot 24 includes imagesrespectively corresponding to a plurality of items. The CPU 16A detectsimages corresponding to the respective items as objects and givesdifferent identification labels to the respective objects to store theobjects. An image processing method for detecting an object in an imageincludes, for example, a method for registering feature data indicativeof a feature, such as a shape and a color, of an object, i.e., detectiontarget, in advance and performing detection based on the feature data.It may include another existing image processing method.

In a case illustrated in FIG. 8(A), for example, objects OB1 and OB2corresponding to beverages (a beer mug and a glass) and objects OB3,OB4, and OB5 corresponding to food (dishes) are detected.

The CPU 16A detects an object image (second object) corresponding to theitem provided to the customer from the second image in a similar mannerto that from the first image (Act S4), and stores the second object inthe RAM 16C or the storage device 16D.

FIG. 8(B) is a diagram conceptually illustrating the second image. In acase illustrated in FIG. 8(B), the objects OB1 to OB5 are detected fromthe second image in a similar manner to that from the first image.However, as illustrated in FIG. 8(B), since the second image is capturedafter the prescribed time elapses from capturing the first image, itemsare consumed by the customer as an eating and drinking progresses. Inaddition, locations at which the objects OB1 and OB2 corresponding tothe beverages (beer mug and glass) are placed on the table are changedby the customer.

To detect changes (remaining quantities) in the states of respectiveitems, the CPU 16A associates the plurality of first objects included inthe first image with the plurality of second objects included in thesecond image (Act S5). Therefore, changes in the state for each item canbe detected even though the locations of the item on the table aredifferent at the time the first image is captured and at the time thesecond image is captured.

The CPU 16A extracts difference by performing comparison for eachcombination of the first object and the second object associated withthe first object (Act S6). The CPU 16A detects a change between thestate of the item captured at the time of capturing the first image andthe state of the item at the time of capturing the second image.

The CPU 16A compares the difference for each combination of the firstobject and the second object with a preset reference value (Act S7), todetermine whether or not the difference exceeds the reference value (ActS8). The reference value is used to determine a state in which the itemis quite consumed, in other words, a state in which the remainingquantity is small or noting remains. The reference value is set todetermine a status in which an item to be subsequently ordered isrecommended. Meanwhile, a part corresponding to the remaining quantityof the item may be detected based on the difference between the firstobject and the second object.

In this case, the reference value is set to determine the remainingquantity, the reference value is compared with the remaining quantity,and thus the state, in which the remaining quantity of the item issmaller than the preset prescribed value or nothing remains, isdetermined. In addition, the reference value may be set for eachcategory of the object (beverage or food) or may be set for each item ifthe item can be specified at the time of detecting the object.

The CPU 16A determines whether or not a notification condition ofproviding the recommendation information to the customer is satisfiedbased on a result of the comparison processing. It is determined thatthe notification condition is satisfied if the first object is comparedwith the second object and then an object (item) of which the difference(result of the comparison) is greater than the reference value exists.The CPU 16A may determine that the notification condition is satisfiedif the difference of at least one object exceeds the reference value orif differences of a predetermined number of objects exceed the referencevalue.

In the above-described comparison processing, the difference between thefirst image (first object) and the second image (second object) isextracted, and the state of the item is determined based on thedifference. However, the state of the item may be determined by anothermethod. For example, feature data of the image captured by photographingthe item is registered for each of the plurality of items in advance asdictionary data, and the dictionary data is compared with the image ofthe respective objects extracted from the second image. Then, it maydetermine whether or not a degree of similarity between the image of theobject and the dictionary data is below a prescribed reference value,that is, whether or not a state of the consumption of the item is quitelarge.

If it is determined that the notification condition is not satisfied (Noin Act B6), the CPU 16A outputs no recommendation information. After theprescribed time elapses from the transmission of the last time imagetransmission request (Yes in Act B2), the CPU 16A transmits the imagetransmission request to the table robot 24 (Act B3), similarly to theabove description. When the CPU 16A receives the imaging data of thesecond image from the table robot 24, the CPU 16A performs thecomparison between the first image and the second image and determineswhether or not the notification condition of providing therecommendation information to the customer is satisfied (Act B5 and ActB6), similarly to the above description.

In the second and subsequent comparison processing, the first image isalready received (Yes in Act S1 in FIG. 7), and the CPU 16A omits theprocessing of detecting the first object from the first image. However,if an additional item is delivered, thereafter, the table robot 24captures the first image of the additional item and transmits theimaging data of the first image (Act A1 and Act A2). In this case, theCPU 16A extracts an object corresponding to the additional item from thefirst image, and stores the additional item similarly to another object.As to the second object disappearing from the second image, it isassumed that the item is fully consumed and collected, and the firstobject corresponding to the second object is also deleted.

If it is determined that the notification condition is satisfied (Yes inAct B6), the CPU 16A outputs recommendation information indicative of anitem extracted by performing the recommendation information extractionprocess to the order reception terminal 22 according to the item ofwhich the state is determined as a state fully consumed (the state inwhich the remaining quantity is small or nothing remains), and notifiesthe table robot 24 of output of the recommendation information (Act B7).

FIG. 9 is a diagram conceptually illustrating the recommendationinformation extracted by the recommendation information extractionprocessing according to the first embodiment.

The CPU 16A performs recommendation information extraction processing30, to register the recommendation information indicative of the item,of which the remaining quantity is determined as small or nothing byperforming the comparison processing, and an item to be recommended tothe customer based on various information 32 stored in advance accordingto the customer in a recommendation menu table 34, an added menu table35, and a last order table 36 stored in the RAM 16C or the storagedevice 16D. The item to be recommended as the recommendation informationincludes, for example, an item recommended in accordance with preferenceof the customer as an additional order according to the item consumed bythe customer, a standard item added in association with the consumptionof the item, a commodity recommended in accordance with preference ofthe customer as a last additional order up to the end time, and thelike.

The various information stored in advance according to the customerincludes, for example, customer information (number of persons, gender,age, job, and the like), today's order history information indicative ofa history of items ordered today, past order history informationindicative of the history of items ordered at the past visiting,information indicative of a today's ordered course menu (prix fix menu),an elapsing time from the check-in processing for the customer, aremaining time up to the end time set in the ordered course menu, or aremaining time until the restaurant is closed, and the like. The pastorder history information can be collected in a case of registering acustomer as a member in advance such that, for example, anidentification data (member ID) for identifying the member (customer) isset before eating and drinking starts, and the ordered items or coursemenu is stored in association with the member ID. The customer inputsthe member ID from, for example, the table robot 24 or the orderreception terminal 22 before he or she orders item, and then, refers tothe recommendation item menu selected from the past order historyinformation on a screen on which the recommendation item menu isdisplayed.

In the recommendation menu table 34, the added menu table 35, and thelast order table 36, for example, at least one item is registeredaccording to a timing at which the item is recommended to the customer.In this case, however, a plurality of items can be registered inassociation with priorities.

In the recommendation menu table 34, for example, an item recommended inaccordance with the preference of the customer is registered as theadditional order of the consumed item. The timing at which the item isrecommended to the customer may be different according to, for example,the age of the customer information. For example, in a case of a childcustomer, eating and drinking time is generally short and thus a dessertor the like usually provided at the end of eating and drinking may beextracted earlier as the recommendation even though the remaining timesufficiently remains. In addition, if a correspondence relationshipbetween the item and the customer is specified, the item to berecommended to the customer may be specified based on the order history(today's order history information or past order history information)for each customer.

An association between the commodity and the customer is performed insuch a way that a location of the item on the table and a location ofthe customer who takes the seat are detected based on, for example, animage captured by the table robot 24 (camera 24G) to determine acorrespondence relationship between the item and the customer.

In the added menu table 35, a standard item added with the progress ofthe consumption of the item is registered. The standard item includes,for example, an item added by a large number of customers, and an itemordered in combination with another item if a course menu is ordered.

In the last order table 36, an item, such as a beverage or a dessertordered at last according to the remaining time up to the end time setin the ordered course menu or the remaining time until the restaurant isclosed, is registered in accordance with the preference of the customer.

When the CPU 22A of the order reception terminal 22 receives therecommendation information from the hall controller 16 (Yes in Act A6),the CPU 22A displays a screen including the recommendation menu on thetouch panel 22G in a pop-up manner (Act A7). In addition, if the hallcontroller 16 notifies the table robot 24 of transmitting therecommendation information to the order reception terminal 22, the tablerobot 24 outputs, for example, a voice message indicating that therecommendation menu is present from the speaker 24F.

FIG. 10 is a diagram illustrating a recommendation menu screen D3displayed on the touch panel 22G of the order reception terminal 22according to the first embodiment.

The menu screen D3 illustrated in FIG. 10 includes, for example, an itemdisplay area D31 in which the recommendation item is displayed, functionbuttons D32 to D35, and a menu selection area D36. In the item displayarea D31, for example, a plurality of recommendation items A, B, C, andD are displayed in such a manner that they are arranged in the order ofpriorities. If the touch operation is performed on one of the areas inwhich items displayed in the item display area D31 are respectivelydisplayed, the CPU 22A determines that the item in the one area isselected.

The function buttons include a TOP button D32 which is used to switch ascreen displayed on the touch panel 22G to a top screen, an order buttonD33 which is used to instruct to transmit an ordered content to theorder station 10, a history button D34 which is used to switch a presentscreen to a screen on which the order history is displayed, a settlementbutton D35 which is used to instruct the POS terminal 12 to perform thesettlement processing, and the like.

The menu selection area D36 includes, for example, buttons correspondingto respective categories such as “recommendation”, “snack”, “beverage &alcohol”, “meal”, and “course (prix fix) menu”. If one of the buttons isselected, the present display is changed over to a screen including amenu of the category corresponding to the selected button.

When the CPU 22A receives an operation on the touch panel 22G (Yes inAct A8), the CPU 22A determines whether or not the instruction forordering the item is performed. If it is determined that the instructionfor ordering the item is performed (Act A9, Yes), the CPU 22A cancelsthe menu screen, and performs processing according to the inputoperation. In contrast, if it is determined that the instruction isother than the instruction for ordering the item (No in Act A9), the CPU22A transmits the order information indicative of the designated item tothe order station 10 (Act A11).

In the above description, the order of the item is received through theorder reception terminal 22. However, the order of the item may bereceived through the table robot 24 with the voice recognition function.For example, if it is determined that the voice is a command forinstructing the order of the item as a result of the recognition of thevoice data input from the microphone 24E, the CPU 24A of the table robot24 recognizes a voice indicative of the item input subsequent to thevoice command. The CPU 24A transmits the item information indicative ofthe recognized item to the order station 10.

As to the recommendation item based on the recommendation informationfrom the hall controller 16, the order of the item recommended may beperformed through more simple voice input. For example, as to therecommendation commodity, the order information may be transmitted byinputting voice of only an item name or inputting voice of a simplemessage indicative of approval (for example, “order” or the like). Inthis way, the order of the recommendation is easily performed and thusthe additional order by the customer can be promoted.

Furthermore, the recommendation item menu may be offered by not onlydisplay of the recommendation item according to the recommendationinformation transmitted from the hall controller 16 but also the voicecommand for instructing the table robot 24 to perform voice recognition.For example, if the table robot 24 performs the voice recognition of avoice command “recommendation menu” for instructing display of therecommendation menu, the table robot 24 requests the hall controller 16to transmit the recommendation information. When receiving the requestfor transmission of the recommendation information, the hall controller16 transmits the recommendation information extracted, for example,through the above-described recommendation information extractionprocessing, to the order reception terminal 22 (or the table robot 24).Therefore, the recommendation item menu can be provided at a timingrequired by the customer.

As described above, in the order processing system according to thefirst embodiment, the eating and drinking situation of the delivereditem (beverage or food), i.e., the progress of the consumption of theitem by the customer (the state in which the remaining quantity is smallor nothing remains) at the time the prescribed time elapses is detectedbased on the captured image, to generate the recommendation informationbased on the item, and then the recommendation information (item) can beprovided to the customer. At the table, the item can be recommended inaccordance with the preference of the customer at the appropriate timingbased on the image captured by the table robot 24 (or the orderreception terminal 22. Even if a plurality of items of which theremaining quantities are small (or nothing) exist in the captured image,different items corresponding to each of the plurality of items can berecommended.

Second Embodiment

FIG. 11(A) is a flowchart illustrating a process performed by the tablerobot 24 and the order reception terminal 22 according to a secondembodiment, and FIG. 11(B) is a flowchart illustrating a processperformed by the hall controller 16 according to the second embodiment.In the description below, Act C1 and Act C2 are described as operationsof the table robot 24, and Act C3 to Act C8 are described as operationsof the order reception terminal 22 in the flowchart in FIG. 11(A).Processing in Act C3 to Act C8 is performed similarly to that in Act CA6to Act A11 according to the first embodiment, and thus detaileddescription thereof is not repeated. In addition thereto, operationswhich are common to that in the first embodiment are not described.

In the second embodiment, a video image of an area (image capturingarea) including the table on which at least item is placed isphotographed by the camera 24G of the table robot 24 (or the orderreception terminal 22), and the hall controller 16 enables the employeeof the restaurant to check the video image.

The table robot 24 performs an image capturing by the camera 24G afterthe item is delivered to the table (Act C1), adds, for example, dataindicative of the table number to an imaging data of the captured videoimage, and transmits the imaging data with the table number to the hallcontroller 16 through the communication device 24H (Act C2).

On the other hand, the CPU 16A of the hall controller 16 manages varioussituations in the restaurant based on the inputs from the handy terminal20, the order reception terminal 22, the table robot 24, and the floorrobot 26. The CPU 16A displays a screen for showing a managementsituation according to the input operation by the employee. For example,if one of the tables is selected on a screen showing the status of theseats, the video image of the selected table can be displayed.

FIG. 12 is a diagram illustrating a screen D1, which shows the seatstatus, displayed on the display device 16E or the tablet terminal 16Kof the hall controller 16.

The screen D1 illustrated in FIG. 12 displays, for example, a switchoverbutton D11, a table layout D12, a floor robot mark D13, a table mark D14(A01 to A10), a table robot mark D15, a communication mark D16, and atable layout D19.

The switchover button D11 is used to instruct switchover of the screen,and includes buttons respectively corresponding to, for example,“cooking status”, “shift management”, and “various tasks” in addition toa button “seat status” on the screen illustrated in FIG. 12. The tablelayout D12 shows the arrangement of the plurality of tables in a displaytarget area in the restaurant. FIG. 12 shows 12 (twelve) tablesindicated with table numbers A01 to A10, B10, and B12 as the displaytarget. The table layout D12 is changed in accordance with the displaytarget area that is changed according to the input operation by theemployee. The floor robot mark D13 is used to indicate the location andthe operation state of the floor robot 26 traveling in the restaurant.In FIG. 12 3 (three) floor robots 26 works in the restaurant. The floorrobot mark D13 is displayed at a location corresponding to the currentlocation of the floor robot 26 which travels in the restaurant. Thetable marks D14 (A01 to A10, B10, and B12) indicate each of theplurality of tables (seat information), and the customer information andthe like is displayed within the mark. The table robot mark D15indicates that the table robot 24 is placed on the table. Thecommunication mark D16 indicates the number of times that the tablerobot 24 communications with the customer. The communication between thetable robot 24 and the customer includes, for example, conversation(response to the output voice message) with the table robot 24, an orderof the item from the table robot 24, and the like. The table layout D19shows the arrangement of all the tables in the restaurant, and specifiesan area in which the table layout D12 is displayed.

If detecting an operation of selecting one of the table marks D14 (ActD1), the CPU 16A of the hall controller 16 displays a detail screen forthe table corresponding to the selected table mark D14 on the displaydevice 16E or the tablet terminal 16K (Act D2).

If the imaging data of the video image captured by the table robot 24 onthe selected table is received (Yes in Act D3), the CPU 16A performsdisplay of the video image on the detail screen based on the imagingdata (Act D4). FIG. 13 is a diagram illustrating a detail screen D2(details of the seat status).

The detail screen D2 illustrated in FIG. 13 includes, for example,customer management information D21, a video image display area D22, anda plurality of function selection buttons D23 to D29. The customermanagement information D21 including the customer information showsinformation (an ordered course menu and remaining time determined forthe course menu) indicative of a current situation of the customer. Inthe video image display area D22, a video image according to the imagingdata received from the table robot 24 is displayed. The CPU 16A changesthe display target area of the video image based on the touch-operationon the video image display area D22. For example, in an initial state, avideo image of an area on the table is displayed among video imagescaptured by the table robot 24. Subsequently, for example, if thetouch-operation is performed on the video image display area D22, avideo image including entirety of the table is displayed. Furthermore,if the touch-operation is performed, a video image including thecustomer who takes the seat on the table is displayed. In this way, theemployee performs an operation to display a video image corresponding toa random arbitrary area and thus can check the situation of the customerincluding the consumption state of the item, from the video image.

The plurality of function selection buttons D23 to D29 include, forexample, a recommendation menu button D23, a menu addition button D24, alast order button D25, an order history button D26, a delivery buttonD27, a bussing button D28, and an OK button D29. The recommendation menubutton D23, the menu addition button D24, and the last order button D25are used to output the recommendation information to the order receptionterminal 22 according to the input operation by the employee. Therecommendation information is registered in each of the recommendationmenu table 34, the added menu table 35, and the last order table 36through the recommendation information extraction processing 30described in the first embodiment. Meanwhile, according to theoperations on the recommendation menu button D23, the menu additionbutton D24, and the last order button D25, a plurality of itemsaccording to the recommendation information corresponding to therespective operations may be displayed on a sub screen in a list form,and a recommendation target item may be selected by the employee. Theorder history button D26 is used to display the order history, forexample, for today in a list form. The delivery button D27 is used toinstruct the floor robot 26 to deliver the item. The bussing button D28is used to instruct the floor robot 26 to travel to a designated tableto collect (bus) tableware, glass, waste and the like from the table.The OK button D29 is used to instruct the end of the display of thedetail screen D2.

If the CPU 16A of the hall controller 16 detects the operation on thebutton provided on the detail screen D2 (Yes in Act D5), the CPU 16Adetermines whether or not the input operation is performed on the buttonfor instructing the output of the recommendation information, that is,one of the recommendation menu button D23, the menu addition button D24,and the last order button D25. If it is determined that the inputoperation is performed on another button (No in Act D6), the CPU 16Aperforms another processing according to the operated button (Act D8).

On the other hand, if it is determined that one of the recommendationmenu button D23, the menu addition button D24, and the last order buttonD25 is operated, the CPU 16A outputs the recommendation information ofthe item corresponding to the operated button to the order receptionterminal 22 on the table selected as the display target of the videoimage, and notifies the table robot 24 of output of the recommendationinformation (Act D7).

Hereinafter, the order reception terminal 22 and the table robot 24operate similarly to that in the first embodiment, and thus detaileddescription thereof is omitted (Act C3 to Act C8).

As described above, in the order processing system according to thesecond embodiment, the employee checks the video image captured by thetable robot 24 provided on each table on the hall controller 16, and cantransmit the recommendation information to the order reception terminal22. In the hall controller 16, the employee arbitrary selects a table onthe screen D1 on which the seat statuses are displayed to display thevideo image in the video image display area D22 of the detail screen D2.Accordingly, the employee checks the consumption state of the item, thecustomer state, and the like for each table, and thus can recommend asuitable item at an appropriate timing.

Third Embodiment

FIG. 14(A) is a flowchart illustrating processing performed by the tablerobot 24 and the order reception terminal 22 according to a thirdembodiment, and FIG. 14(B) is a flowchart illustrating processingperformed by the hall controller 16 according to the third embodiment.In the description below, Act E1 and Act E2 are described as operationsof the table robot 24, and Act E3 to Act E8 are described as operationsof the order reception terminal 22 in the flowchart in FIG. 14(A).Processing in Act E3 to Act E8 is performed similarly to that in Act CA6to A11 in the first embodiment, and thus detailed descriptions thereofare not repeated. In addition, processing in Act F1 to Act F4 in FIG.14(B) is performed similarly to that in Act D1 to Act D4 illustrated inFIG. 11(B) in the second embodiment, and thus detailed descriptionsthereof are not repeated. In addition, processing common to that in thefirst embodiment or the second embodiment is not described.

In the third embodiment, similarly to the second embodiment, theemployee of the restaurant can check the video image captured by thecamera 24G of the table robot 24 (or the order reception terminal 22),on the hall controller 16, and, similar to the first embodiment, therecommendation information can be output according to a result of thecomparison processing based on the video image received from the tablerobot 24 as well.

In the third embodiment, the first image is extracted from the videoimage received from the table robot 24, for example, after the item isdelivered and is stored. Similarly to the first embodiment, ifdetermining that the prescribed time elapses (Yes in Act F5), the CPU 16of the hall controller 16 extracts the second image from the video imageand performs the comparison processing of comparing the first image withthe second image (Act F6). The comparison processing is performedsimilarly to the first embodiment, and thus detailed description thereofis omitted.

If it is determined that the notification condition of providing therecommendation information to the customer is satisfied based on theresult of the comparison processing (Yes in Act F7), the CPU 16A outputsthe recommendation information to the order reception terminal 22 andnotifies the table robot 24 of output of the recommendation information(Act F8), similarly to the first embodiment.

As described above, in the order processing system according to thethird embodiment, the employee checks the video image captured by thetable robot 24 on each table, on the hall controller 16, and a change inthe state of the item (the state in which the remaining quantity issmall or nothing remains) based on the video image is detected. Thus,the layout information can be automatically output without requiring theoperation by the employee. Therefore, a suitable item can be recommendedat an appropriate timing without applying operational loads to theemployee.

Meanwhile, in the description according to the above-describedembodiments, the image capturing is performed by the camera provided inthe table robot 24 or the order reception terminal 22. However, one ormore cameras are provided in the vicinity (including an upper part) ofthe table separately from the table robot 24 or the order receptionterminal 22, and the image capturing may be performed in an area, forexample, including the table and the customer positioned in the vicinityof the table as an image capturing area.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the invention. Indeed, the novel methods and systems describedherein may be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the methods andsystems described herein may be made without departing from the spiritof the invention. The accompanying claims and their equivalents areintended to cover such forms or modifications as would fall within thescope and spirit of the invention.

In addition, in the processes described in the above-describedembodiments, it is possible to write a program which is capable of beingexecuted by the computer in, for example, a recording medium, such as amagnetic disk (a flexible disk, a hard disk, or the like), an opticaldisk (a CD-ROM, a DVD, or the like), or a semiconductor memory, and toprovide the program to various devices. In addition, it is possible totransfer the program using a communication medium and to provide theprogram to the various devices. The computer reads the program recordedin the recording medium or receives the program through thecommunication medium, and performs the above-described processes in sucha way that an operation is controlled by the program.

What is claimed is:
 1. A monitoring-based service providing systemcomprising: a table monitoring terminal associated with one of aplurality of tables in a venue and including: a camera; a drivingmechanism configured to change a direction of the camera; a firstcommunication interface; and a first controller configured to controlthe driving mechanism such that the camera is directed to at least aconsumable item on an associated table, and control the firstcommunication interface to transmit image data of an image including aconsumable item captured by the camera; a central control terminalincluding: a second communication interface communicable with the firstcommunication interface; and a second controller configured to: comparethe image data received by the second communication at first timing withthe image data received by the second communication interface at secondtiming after the first timing; determine a remaining amount of theconsumable item based on a comparison result of the image data receivedat the first and second timing; and generate recommendation informationupon the determining remaining amount decreasing to a predeterminedthreshold, and control the second communication interface to transmitthe generated recommendation information; and an order terminalassociated with said one of the plurality of tables and including: athird communication interface communicable with the second communicationinterface; a display; and a third controller configured to control thedisplay to display a recommendation screen based on the recommendationinformation received by the third communication interface.